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often by special education teachers in their evaluation of student 
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and that they are confident in .their assessments o{ student 
performance. Observations of teachers during instruction corroborated 
these survey findings; teachers actually relied on and were confidentj, 
in their informal observations of student performance on lesson 
objectives. Nevertheless, the observed teachers were highly ^ 
inaccurate in their assessments of student mastery of lesson 
objectives and in. their estimates of performance on lesson 
objectives. Implications .for monitoring student progress are 
discussed. (Author/DB) 



* Reproductions supplied by EDRS are the best that can be made * 

* " from the origina;L document. ' * 
***************************************************** 



ERLC 



ED 224 197 

AUTHOR 
TITLE 

INSTITUTION 

SPONS AG^CY 

REPORT NO 
PUB DATE 
CONTRACT 
NOTE 
PUB TYPE 

» EDRS PRICE - 
^ DESCRIPTORS 



urn University of l\/l/nnesot!a 



f 

Research Report' >|q!,f 81 



U S DEPARTMENT OF EDUCATION 
NATIONAL INSTITUTE OF EOUCATION 

tOUCA^jOl^Al HtSOURCES II^ORMATION 

fe\t'i\,»'<i I'o'-i' tut' nt>f'>0(t o' oHjdni^jtion 

• P<i,trs oU'VA uf i,'i> nn.>n») bid 'n this <io<u 
mo'it ijn nut nc< rsvirtly ft>pM*v«'nt of<K t<jl Nil 
portion o( V > 



SPECIAL EDUCATIpN PRACTICE IN EVALUATING STUDENi 

PROGREsffblfiARDGOALS^ - 

/'-. , 

Lynn S. Fuchs, Douglas Fuchs, and Linda M. Warren 





Disabilities 



y 



-PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



2 



(3* 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



9 ' . <J 



1.^ 



Director: James E. Ysseldyke 



The Institute for Research on Learning Disabilities is supported by 
a contract (300-80-0622) with Special Education Programs ^ Department 
of Education. Institute investigators ane conducting research on the 
assessment/decision-making/intervention process as it relates to learn- 
ing disabled s.tudents. 

» 

During 1980-1983, Institute research focuses on four major areas: 
t Referral 

• Identi fication/Classi fi cation. 

• Intervention Planning and Progress Evaluatfion 

f. Outcome Evaluation 

Additional information on the Institute's research o'bjectives and 
activities may be obtained by writing to the Editor at the Institute 
(see Publications list for address). ' ^ 



The -research reported herein was conducted under government spon- 
sorship, 'contractors are encouraged to express freely their pro- 
fessi>onal judgment in the conduct of the project. Points of view 
or opinions stated do not, therefore, necessarily represent the 
official position of Special Education Programs; 



Research Report No. 81 

SP,ECIA1- EDUCATION PRACTICE IN EVALUATING STUDENT 
PROGRESS TOWARD 'GOALS 

« 

^ Lynn's. Fuchs 

Institute for Research on Learning Disabilities 
University of Minnesota 

Douglas Fuchs 
I Clark University , 

Linda Warren 
Wheel ock College ^ 

• ^ ^ July, 1982 



Abstract 

• Surveys of 147 special education teachers and observations of 20' 
practicing teachers And 20 cooperating teachers were used * to (a) 
determine which evaluation procedures are used most often by special 
education teachers in .their evaluation of student progress, and (b) 
assess the adequacy of those procedures. Survey respondents indicated 
that typically they assess progress on lEP objectives quarterly, they 
reTy on informal observation for assessing students' mastery of 
objectives, and they are confident in their assessments of student 
performance. 'Observations of teachers during instruction corroborated' 
the$e survey findings; teachers actually relied on and were confident 
in their ; informal observations of student performance on lesson 
objectives. Nevertheless, the, observed teachers were highly 
inaccurate irt their assessments of sttrdent mastery^ of Tesson 
objectives and in their estimates pf performance on lesson objectives. 
Implications for monitoring "student progress are discussed. 



Special Education Practice in Evaluating Student 
Progress Towards Goals 

PL 94-142 requires special education teachers to formulate shortJ- 
term objectives and annual goals to ensure appropriate education and 
to facilitate handicapped pupils* movement toward less restrictive 
educational settings. • Federal law also 'directs special- education 
teachers- to assess* student progress towarcf specified objectives and ' 
goaK. Despite this apparent concern ' for and emphasis on student 
evaluation, PL 94-142 does not specify how student progress should be 
assessed; educators are free to choose whatever assessment procedures 
they wish. . y^'^^ 

A reasonable as^ti^ff^ti'on is that *=**^$roecial educators ' employ a 
varij?tyoT approaches to evaluate 'pupil progress. However, 4..,precise . 
description of these approaches ^^nd the frequency of their use 
currently is unavailable. Without such information, it is impossible 
to determine the adequacy of^special education practice in monitoring 
pupils* progress toward lEP goals and objectives. The purpose of this 
investigation was to address two questions: (L) Which evaluation 
procedures are used most often by" special' education teachers to 
evaluate student progress, and (2) How valid are these procedures? 

" . • Method ' 

Survey L 

Subjects . A survey was mailed to 4O0 special educators. Twenty 
were special education * teachers who, when the ^ study began, ha*d 
trainees in their classrooms from a local teacher-training college. 



The remaining 380 special^ educators were the non->student member ^ 
population of the Massachusetts Federation of the Council for 
Exceptional Ch>ldren (CEC), Of the total group, 147 (ST^^^^turned 
their completed surveys' within^ a^specified time ,limit. ihis response 
rate^ref lects the fact that many. of the CEC niembers to whom the s'Orvey 
was were*-oot -teachTers^l the^ cover Jetter 'reque^tefd that- only 

te/chers complete the survey form» ^ • ^ . • 

The responding teachers (19 male, 128 female) had' taught special 
education an average 8,09 y^ars (SD = 5.27), with 49.6% curt/ently 
conducting mainstream or resource ^programs, .27.9% teciching special 
-education self-contained public school classrooms,^ and 22^4% wbrking 
in other setting^ including special education sVl^-corrtained private 
or resi^ientia^l schools, private ^ cl inics^ vocational educational 
schools* preschool s*pecial education centers. The highest 

educational degree earned was a bachelor's degree for 35 teachers 
(23.8%), a master's degree for ^82 teachers (55.8%), two master's 
degrees for 16 teachers (10.8%), and a specialist or doctoral' degree 
for 14 teachers (9:5%). All ^ subjects were certified as special 
educators . 

Materials and proceclure . A survey was developed to investigate 
how special educators assess their student^ ' mastery of both' lEP 
objectives and instructional material 'presented in daily lessons (see 
Survey Form in Appendix). In addition to requesting background 
information, the survey contained four items that .explored: (a) how 
'often evaluations are conducted on pupils' progress toward the.ir lEf 
objectives, (b) the types of measures that are ased to assess^ this 



progress, (c) ^K'ow confident special educators are in their estimates 
1 ^ * 

of students' daily performance on instructional objectives, and (d) 
p^rocedures for assessing students^ level of performance on objectives 
addressed during daily lessons. Surveys and stamped return envelopes 
were mailed in May 1982. No follpw-up contacts were made. 
Observation 

Subjects . Subjects were (a) 20 teacher trainees irh special 
education who were completing their final practicum, and (by 20 
special educators (cooperating teachers), ^ch of whom had a trainee 
in the classroom. Of these cooperating teachers, 12 were in resource 

programs, 6 were in self-contained public school classrooms, and 2 

were in private school 'settings. The highest educational, degree 
earned was a bachelor's degree for 35% of the cooperating teachers, 
one master's degree for 55% of the teacher's^, * two master's degrees- for' 
5%, and a specialist for 5%. These- teachers -had taught.- special ^ 
eduoatiort for an average 7;^33 years (SO = 4.17) and were - properly.' ' 



recertified by the state of Massachusetts. All trainees and cooperating ' 
teacners were female. = - * 

' PrWedure. Each trained -was placed i^n a 231 hour; practicum with 

•a cooperating teacher. Approximately *16() hours^into this, practicum, 
the. cooperating teacher and one of two observers, wHo had been trained 
in the study procedures,* watched ^the trainee conduct a less6n with a 
pupil who was ^nralled Jn the cooperating teacheV's program. 

- Prior; to \iis observation, the trainee provided the observer and 
the cooperating tQacl^r with a lesson plan, and a. behavioral objective 
that. set a criterion of performance in terms of percentage correct 



(see Ob$erva.tt^r\,%J^eport Form, Part A', in^ Appe,ndfx) / Durina the 
lessopf; • the '.obsenver recorded the child*s actual performance on the 
behavioral, bbjecti^ve (percentage 'correct) and pescribed the methods 

.. ■ ■ ■ ^^1 

employed J^ry the** trainee to "assess the child'i performance ,on the 
,behavior^.l .objec^ti-ve- (see Supervisor , Observation Form in Appendix). 
Follwin^^ the lesson, the' trainee and -the C9operating teacher 
inde'pendently (a) rated the success of the* lesson, (b) provided a 
rationale for that rating, (c) indicated whether the student had 
mastered the behavioral ^objective, and (d) if' the child hay not 
mastered the .objective, es'ttfhated ' the actual level of performance 
(percentage cp'rrect) on" tbe objective (see Observation Report Form, 
Part'B^ in Appendix)-. Thjs procedure was repeated for each teacher 
-trainee and cooperating-'teacher^^ 

Inter-rater agreement' for* vfche observers* meas'urements of 
Chi Idren's • actual ''performance on behavioral objectives was assessed 
across 5 of the 20 observations. Reli'a&iTity coefficients (proportion 
of agreement) ranged from .89 to- l..Q^, With an average coefficiept of 
.94. 

Data analysis .' The accuracy o^ the trainees' and cooperating 
teachers' estimates of child performance on the behavioral objective 
were compared^usi ng _t tests/ AccuracW was defined as^ the absolute 
value of the difference between the actual observed percentage 'and the 
estimated percentage (see Observation Report Form, Part B).^ Low 
scores indicated ^greater accuracy than -mgh .scores/ Teachers were 
awarded a s-cor^ of 0 when they, correctly determined that-an objective 
had been mastered, A corre/lated t^ test wa$ conducted to determine 



whether the inaccuracy of teachers/ estimates was dependent on the 
level of teacher experience -(trainee vs. cooperating teacher). Chi 
square analyses also were run to determine whether teacher accuracy 
(high, vs. low) was related either to the^ behavioral objective actually 
being met or'to the teachers' judgments that the objective had been 
met. Additionally, criteria used to determine thg. success of lessons 



and the trainees' measurement procedures were summarized/ 




Results 

Survey ' \ ' • 

Question 1 on the survey asked, how frequently teachers determine 
whether students' lEP objectives have been achieved. Four teachers 
(2.75^) responded . annually, 14 (.9.5%) weekly, 15 (10.2%) at periodic 
review, 97 (65.9%) quarterly, 6 (4.1%) semi-annually, and 11 (7.5%) 
responded in other ways, including 5 (3.4%) daily, 3 (2.0%) monthly, 
and 3 (2.0%) no response. 

The second question required respondents to check al 1 of/ the 
procedures they .use to' determine ' whether students have met lEP 
ob^jectives, and then ^to circle the one procedure they rely on the most 
heavily to determine .wh^er IE? ^objectives hav^ been ^mastered. 
Informal observation was marked by 96 -of the respondents ,(65.3%), 29 
(19.7%) indicated^ criterion-referenced ^instruments; 9 (6.1%) marked 
norm-referenced standardized tests; and 13 (8.8%) wrote teacher tests 
or teacher-made assignments. Of the 96 respondents who reported thit 
they primarily employ informal observation, 30 indicated that they 
rely completely on informal observation for determining whether lEP 
objectives have been met. 



The survey's third 'question asked teachers how certain they are, 
in general, about the student's level of performance at the end of an 
academic lesson. Fifty-nine (40.1%) checked* very sure,. 79 (53.7%) 
marked sure, 4 (2.7%) indicated somewhat unsure, and 2 (1.4%) checked 
not sure (three teachers did not respond to' this question) . . 

Question 4 asked how, in the ' majority of cases, teachers 
determine a student's level of performa^nce on the material covered in 
a lesson. Approximately 80%^ of the respondents checked "observing 
informally during a lesson, 5.4% stated that they administered a 
test, and 15.0% indicated that they employed a teacher-made exercise. 

When responses of the 20 cooperating teachers who participated in 
the observation phase of this 5"tudy were analyzed separately, 15% 
indicated that they assess mastery, of lEP objectives at periodic 
review, 70% quarterly, 5% daily, and 10% semi-annually. With respect 
to question 2, 55% reported that they rely predominantly on informal 
observation, 10% on criterion-referenced measurement, 10% on norm- 
referenced standardize^d' tests, and 15% on teacher-created exercises. 
For material covered in daily lessons, 45% were very sure of the 
accuracy with which they assess students' level of performance; 55% 
were sure. To assess level of perforrliance on material covered in 
daily -lessons, 85% of the cooperating teachers reported that they rely 
on inf/ormal observation, 5% indicated tests, and 10% marked that they 
employ their own exercises. • . 

Observation ' ^ / 

Measurement procedures employed . Of the 20 trainees, 3 measured 
children's performance on the lesson objectives with' written products. 



either a worksheet or a more [forflia>l test. SixtVen trainee^ used games 
or flashcards to measure the students' performance, but^'none of these 

V 

trainees recorded pupi 1 s ' correct and incorrect responses. One * 

trainee did not attempt to measure in any 'fashion the child's 

performance on the behavioral objective. (The observer independently 

tested the child following that lesson to determine actual level of 

w 

performance.) * ' 

' Accuraicy of estimates of students' performance . A t_ test 
revealed that the trainees' estimates of student performance and 
students* actual . performance on behavioral objectives was 
significantly different from zero, (19) = 5.17, £ < .001 (mean = 
15.45/t). Cooperating teachers' estimates also were signif icnatly 
different from zero, _t (19) = 4.61, £ < .'001 (mean = 15.65%). Thus, 
both travnees. and^cooperating teachers were inaccurate in their 
estimates of the children's performance. A ^correlated t_ test 
conducted on the difference between the trainees' and cooperating 
teachers' estimates indicated th'at there was no^ relation, between 
teaching experience and' degree of .accuracy in estimating levels of 
performance. 

Both trainees and cooperating teachers more frequently judged 
that objectives had been met rather than failed (see Table 1). 
However, a chi square analysis revealed no'relation between teachers' 
accuracy (high < |10|; low > \10\) and their tendency to judge whether 
objeativeS had been met. 



Insert Table 1 about here 



Nevertheless, there f^as a relation between trainees' and 
^teachers* accuracy and chilldren's actual mastery and non-mastery of 
objectives, x^(l) = 5.03, £ < .025 for trainees, and x^(l) = 5.28, £ < 
.025 for teachers. When pupils achieved the instructional objectives, 
trainees' and cooperating teachers' judgments alw&ys were accurate; 
when obj'ectives were not met, evaluations tended to be inaccurate (see 
Table 2). 



Insert Table 2 ^about here 



Judgments of succes^S . Trcfinees and cooperating teachers judged 
lesions as either very successful or successful. Among the trainees,- 
four rated their lessors as very successful , and 16 as successful. 
Eight of the' cooperating teachers judged the - lessons .as very 
suctessful and 12 as successful* ^ Teaching experience was not; 
significantly , relatexl to the teachers' judgments of the lessons' 
succeS^. 

Seven of the trainees were very sgre about their judgments of. 
success, 12 were sure, and one was somewhat un$ure. Of the 
cooperating teachers, 13 were very sure of their judgments, and seven 
were sure. The teachers' experience was related to the certainty with 
which they rated the success of lessons, x^(l) = 5.14, £< .025. 

Trainees most frequently cited some aspect of the child's 
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performance, as evideno^, ' supporting their judgments of lessons* 
success. Lfesson completion, rapport with or control of-the child, and 
the child's enjoyment of t)ie lesson also were mentioned as reasons to 
explain ratings of the^ lessons* ^supcess. In contrast, the coopeVating 
teachers ^most frequently referred to some aspect of the instructional 
content of the lesson to. explain their ratings of succ.ess. 
Cooperating teachers also named flexibility, organization, and the 
child's performance. / » ^ . ' 

Discussion ; 
The purposes of the study were to determine what procedures 
.special education" teachers eofriploy in their assessment of student 
performance on goals and objectives, and to investigate the adequacy 
of those procedures. A total of 147. special education teachers 
responded to a survey designed to gather information on procedures 
employed by special educators to assess performance on lEP and daily 
objectives. 

• Findings indicate- that ' mojst special education teachers (65.9^) 
evaluate progress on lEP objectives approximately four times yearly, 
and that a majority^ (65.3%) rely on . informal observations compi led 
over each quarter to formulate their-, decisions conceVj^ing whether lEP 
objectives have been met* - * 

Similarly, for assessing a^ student' s . level of performance Ss^ 
material covered in daily less.ons, teadhers 'reported overwhelmingly 
that they employ .informal observation (80.0%) and indtcated they ^re 
qurte confident about the accuracy of assessments formulated on 
informal observations. 



The survey data were corroborated by observational datci. 
Observations of practicing teachers, who were implementing lessons for 
which they .had' written behavioral objectives^ indicated that 17 of 20 
{8S.0%^ actually relied on informal observation to assess the 
students' level /Of performance. Ninety-five percent ^pf the observed 
practicing teachers and 100% of their cooperating te^ciiers were very 
sure of. the accuracy of their judgments of the' lessons' success. 
Thus, ilt •appears /that special education ' teachers tend , to employ 
informal observation to formulate their decisions about students* 
performance on objectives and-do so with confidence about. the accuracy 
of those assessments.. 

^ * I? 
However, additional data suggest that the confidence teachers 

place in the accuracy of their informal observations may not be well 

founded. Among 20 practicing teachers and 20 cooperating teachers, 

who were representative of the su>:vey sample in terms of experience, 

training, ' and special program typ^, the accuracy of assessments of 

childi^en's performance on objectives .was highly inaccurate. Although 

> 

teachers recognized when objectives actually had been achieved, they 
failed to judge accurately when objectives were not met. Ror children 
who actually had ..failed' objectives, practicing and cooperating 
teachers frequently indicated that objectives had been met. When they 
correctly recognized th&t the objectives had not been achieved, the 
practi cing and cooperati ng teachers were, highly inatcurate in their 
estimates of the st|jdents' actual level of performance. In spite of 
this inaccuracy, trainees and cooperating teachers^^aid that they were 
very .sure or sure of their .as^sessments . Ad,ditionally, teaching 
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experience affected neither the accuracy nor the confidence of these 
.assessments. ^ * ^ / * ' 

: It appears, then, that special educ,^tors' reliance on and 
confidence in informal observation as Jan assessment tool is 
unjustified. The purposejif writing instructional objectives in lEP 
and daily lessons is twofo,ld: to help 'teachers structure a child*s 
eiiucation and to assist educators in eA/aluating whether the student 
has changed in the intended ways (Bloom, Hastings, & Madaus, 1971). 
,Qri'tei^ion -referenced assessment is the process of evaluating whether 
^those o()jectives have been met. To the extent to which the. study 
, ^sarTlple,Js representative, this investigation indicates that special 
. education teachers perl^orm criterion-referenced assessments; that^is, 
they fprmulate decisions concerning whether objectives have been 
achieved. However, the study also demonstrates that teachers 
typically do not use systematic procedures to measure children's 
behavior. Rather, special education teachers tend to rely on informal 
observation, a practice that often leads to erroneous judgments of^ 
^levels of , academic performance and , inaccurate conclusions . about 
whether objectives ha(ve been met. ^ v 

These findings suggest that special education teachers should 
reevaluate the role of ijiformal 'observation in their assessment of 
students' performance on object i ves . More systematic measurement 
procedures would: (a) produce data characterized by greater 
objectivity' and < accuracy; (b) lead more often to correct decisions 
about students* realization of objectives; (c) lead ultimately to 
better instructional decisi.ons and student achievement (Mirk in, Deno, 
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Tindal, & Kuehnle, 1980); and (d) address more; adequately the lEP 
component of PL- 94-142 (Deno & Mirkin, 1977). 
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Table 1 

Practicing and Cooperating Teachers with High or Low Accuracy Scores 
Judging that Objectives Were Met or Failed 



Practicing Teachers 
* Judged fudged 
Objective , Objective 
Met Failed 



Cooperating Teachers • 
. Jud^a^ '. Judged 

Objective Objective 

Met Failed 



High , 
Accuracy 
(X < |10|) 



6^(0^ O'') 



1(1.00, 0) 



6(1.63, 3.55) 



Low 



Accuracy-, ¥5(25.00, ^.36) 3(19?33, 3.06) 
CX > Moll 

—7 ^ 



Number of teachers. 

^Mean accuracy score In terms of percentage. 
'Standard deviation of accuracy scores. 



'9(2*8.63, 12.73) 4(17.75., 8.58) 
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Table 2 



Practicing and Cooperating Teachers with High or Accuracy Scores 
When Children Actually Met or Failed Objectives 



High 

Accuracy 
(X < tiol) 



Practicing Teachers 
Child Actually Child Actually 
Mer Objective Failed Objective 



Cooperating Teachers 
Child Actually Child Actually 
Met Objective Failed Objective 



' 6^o^ 



1(1.00. 0) 



6(0, .0) 2(6.50, 4.95) 



Low 

Accuracy 
(X > 110|) 



13(14.00, 8.71) 



12(13.00, 12.'34) 



Number of teachers. 

'^Mean accuracy score in terms of percentage. 
^Standard deviation of accuracy scores. 
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SURVEY FORM 



How frequently do you determine whether students' lEP objectives 
have been achieved? , ' ^ 



jAnnually 
JWeekly 



_At periodic review 
Other 



Quarterly 



(specify) 



First, check (\/)^ the procedures (s) you use to determine whether 
your students have met lEP objectives. Second, circle the one 
procedure you rely on the most to determine whether lEP 
objectives have been met- , ^ 



_Norm-ref erenced 
standardized tests 

_Criter ion-referenced 
tests 



^Informal observations 
Other 



(specify) 



(a) Generally speaking, at the end of an academic lesson, how sure 
are you about the student's level of performance on material 
in that lesson? 



Very Sure 



Sure 



Somewhat 
Unsure 



Not Sure 



(^) In th.e majority of cases, ^how do you deter^ne the student's 
level of performance on the material in the lesson? 



_;^observing informally during a lesson 
administering a test 



other 



(specify) 

Please provide the following information.^ 

Sex; M P Years teaching special education^ 



Program type (i.e., LD resource program, MR self-contaofped, etc.) 



Degree (s) and Certification (s) Earned 



THANK 
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OBSERVATION REPORT FORM (PART A) 



Student Teacher's Name: 

Pupil's Name: 

Date: 

Observation Number: 
Behavioral Objective: 



Please attach your* lesson plan. 
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SUPERVISOR OBSERVATION FORM 



Child performance level on the behavioral objective: 



Describe briefly how the practicing teacher assessed the child' 
performance level on the objective. 




OBSERVATION REPORT FORM (PART B) 



Rate the success of the lesson: 



Very / Successful 
'Successful 



Somewhat 
Unsuccessful 



Not 
Successful 



How sure are you? 



Very 
Sure 



,Sure 



Somewhat 
Unsure 



Not 
Sure 



Briefly explain why you selected the above category of success, 



Did the child master the behavioral objective? 



Yes 



No ' 



If .v"no," estimate the percentage correct at which the child did perf^orm. 
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